Information Geometry of Contrastive Divergence
نویسندگان
چکیده
The contrastive divergence(CD) method proposed by Hinton finds an approximate solution of the maximum likelihood of complex probability models. It is known empirically that the CD method gives a high-quality estimation in a small computation time. In this paper, we give an intuitive explanation about the reason why the CD method can approximate well by using the information geometry. We further propose an improved method that is consistent with the maximum likelihood (or MAP) estimation, while the CD method is biased in general.
منابع مشابه
Information Measures via Copula Functions
In applications of differential geometry to problems of parametric inference, the notion of divergence is often used to measure the separation between two parametric densities. Among them, in this paper, we will verify measures such as Kullback-Leibler information, J-divergence, Hellinger distance, -Divergence, … and so on. Properties and results related to distance between probability d...
متن کاملDifferential Contrastive Divergence
We formulate a differential version of contrastive divergence for continuous configuration spaces by considering a limit of MCMC processes in which the proposal distribution becomes infinitesimal. This leads to a deterministic differential contrastive divergence update — one in which no stochastic sampling is required. We prove convergence of differential contrastive divergence in general and p...
متن کاملStochastic Gradient Estimate Variance in Contrastive Divergence and Persistent Contrastive Divergence
Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) are popular methods for training Restricted Boltzmann Machines. However, both methods use an approximate method for sampling from the model distribution. As a side effect, these approximations yield significantly different biases and variances for stochastic gradient estimates of individual data points. It is well known tha...
متن کاملGeometry-Contrastive Generative Adversarial Network for Facial Expression Synthesis
In this paper, we propose a geometry-contrastive generative adversarial network GC-GAN for generating facial expression images conditioned on geometry information. Specifically, given an input face and a target expression designated by a set of facial landmarks, an identity-preserving face can be generated guided by the target expression. In order to embed facial geometry onto a semantic manifo...
متن کاملUnifying Non-Maximum Likelihood Learning Objectives with Minimum KL Contraction
When used to learn high dimensional parametric probabilistic models, the classical maximum likelihood (ML) learning often suffers from computational intractability, which motivates the active developments of non-ML learning methods. Yet, because of their divergent motivations and forms, the objective functions of many non-ML learning methods are seemingly unrelated, and there lacks a unified fr...
متن کامل